Comparing MapReduce and pipeline implementations for counting triangles

A common method to define a parallel solution for a computational problem consists in finding a way to use the Divide and Conquer paradigm in order to have processors acting on its own data and scheduled in a parallel fashion. MapReduce is a programming model that follows this paradigm, and allows for the definition of efficient solutions by both decomposing a problem into steps on subsets of the input data and combining the results of each step to produce final results. Albeit used for the implementation of a wide variety of computational problems, MapReduce performance can be negatively affected whenever the replication factor grows or the size of the input is larger than the resources available at each processor. In this paper we show an alternative approach to implement the Divide and Conquer paradigm, named dynamic pipeline. The main features of dynamic pipelines are illustrated on a parallel implementation of the well-known problem of counting triangles in a graph. This problem is especially interesting either when the input graph does not fit in memory or is dynamically generated. To evaluate the properties of pipeline, a dynamic pipeline of processes and an ad-hoc version of MapReduce are implemented in the language Go, exploiting its ability to deal with channels and spawned processes. An empirical evaluation is conducted on graphs of different topologies, sizes, and densities. Observed results suggest that dynamic pipelines allows for an efficient implementation of the problem of counting triangles in a graph, particularly, in dense and large graphs, drastically reducing the execution time with respect to the MapReduce implementation.Peer ReviewedPostprint (published version

NiMo syntax: part 1

Many formalisms for the specification for concurrent and distributed systems have emerged. In particular considering boxes and strings approaches. Examples are action calculi, rewriting logic and graph rewriting, bigraphs. The boxes and string metaphor is addressed with different levels of granularity. One of the approaches is to consider a process network as an hypergraph. Based in this general framework, we encode NiMo nets as a class of Annotated hypergraphs. This class is defined by giving the alphabet and the operations used to construct such programs. Therefore we treat only editing operations on labelled hypergraphs and afterwards how this editing operation affects the graph. Graph transformation (execution rules) is not covered here.Postprint (published version

Exploiting parallelism by customizing evaluation

NiMo is a totally graphic language from the family of Higher Order Typed languages with a strong Data flow inspiration. The interpreter is a specialized graph transformation system, and therefore the language operational semantics is given in terms of graph transformations. In NiMo parallelization is implicit and the evaluation policy is customizable following a process-centered approach. Here we explore some of the methodological possibilities that it opens. Some classical examples illustrate how combining modes greatly increases processor usage, decreases channel population, and achieves subnet synchronization in a very easy and intuitive way. We also present a stream programming technique and a real case application for generative and multistage-programming.Peer ReviewedPostprint (author’s final draft

MapReduce vs. pipelining counting triangles

In this paper we follow an alternative approach named pipeline, to implement a parallel implementation of the well-known problem of counting triangles in a graph. This problem is especially interesting either when the input graph does not fit in memory or is dynamically generated. To be concrete, we implement a dynamic pipeline of processes and an ad-hoc MapReduce version using the language Go. We explote the ability of Go language to deal with channels and spawned processes. An empirical evaluation is conducted on graphs of different size and density. Observed results suggest that pipeline allows for the implementation of an efficient solution of the problem of counting triangles in a graph, particularly, in dense and large graphs, drastically reducing the execution time with respect to the MapReduce implementation.Peer ReviewedPostprint (published version

NiMoToons: a totally graphic workbench for program tuning and experimentation

NiMo (Nets In Motion) is a Graphic-Functional-Data Flow language designed to visualize algorithms and their execution in an understandable way. Programs are process networks that evolve showing the full state at each execution step. Processes are polymorphic, higher order and have multiple outputs. The language has a set of primitive processes well suited for stream programming and supports open programs and interactive debugging. The new version of the environment NiMo Toons includes: an also graphic and incremental type inference system, multiple output processes as higher order parameters, symbolic execution, five evaluation modes that can be globally or locally set for each process and dynamically changed, and facilities to measure the used resources (parallelism level, number of steps, number of processes, etc.)Postprint (author’s final draft

Dynamic pipelining of multidimensional range queries

The problem of evaluating orthogonal range queries efficiently has been studied widely in the data structures community. It has been common wisdom for several years that for queries containing more than 20% of the elements of the dataset a linear scanning of the data was the most efficient solution. In recent experimental works using modern hardware –with main memory and parallelism– the conclusion is that linear scan is preferable for almost every query configuration (even containing a 1% of the data). In this work we propose an alternative approach to evaluate multidimensional range queries based on the dynamic pipeline paradigm –using main memory and concurrency. Our aim is to prove that under this framework, it is possible to beat the performance of linear scanning by the one of hierarchical multidimensional data structures –such as kd trees, quad trees, Rtrees or similar.Peer ReviewedPostprint (published version

Graphical type inference: A graph grammar definition

We present a graph grammar based type inference system for a totally graphic language inspired in the data flow view of lazy functional programs. NiMo (Nets in Motion) can be seen as a graphic equivalent to Haskell that acts as an on-line tracer and debugger. The user not only sees the results but also the way they are calculated according to an understandable model and can interrupt the execution at any point, change data, processes and/or process activation, undo steps, and also execute incomplete programs. Type inference is incremental; during the net edition (construction or modification) only type safe connections are allowed. The user visualises the type information evolution and, in case of type error, can identify where and why it happened. The NiMo type system, though similar, has significant differences with systems in functional languages due to the data flow ingredient. It needs to cope with processes with no entries and zero or more that one output and therefore the process type is a generalization of functional types. We present the notion of non-structural type unification, the elements for modelling graphic type inference, and the correspondence with the classical type inference approach. Construction and execution of NiMo programs are fully defined via an attributed graph grammar. In the previous version type information was incomplete and static type inference was partial in presence of polymorphism. Therefore type inconsistent nets could be executed. Here we present the type descriptor graphs and the graph grammar definition of the complete static type inference system. The grammar has been implemented and successfully tested using AGG as the graph transformation system.Postprint (published version

Implementing static synchronous sensor fields using NiMo

In this work we present some implementations of a Static Synchronous Sensor Field (SSSF), a static and synchronous model for sensor networks in which a finite set of sensing devices are geographically distributed and can communicate among them. We used for implementations the NiMo (Nets in Motion) programming language, a graphic-functional-data flow language that allows a step by step visualization of the executions of an algorithm, making visible all the involved elements. Experimenting with sensor networks gives an insight of their behavior and aids to see their properties.Postprint (published version

Mutational spectrum in a worldwide study of 29,700 families with BRCA1 or BRCA2 mutations.

The prevalence and spectrum of germline mutations in BRCA1 and BRCA2 have been reported in single populations, with the majority of reports focused on White in Europe and North America. The Consortium of Investigators of Modifiers of BRCA1/2 (CIMBA) has assembled data on 18,435 families with BRCA1 mutations and 11,351 families with BRCA2 mutations ascertained from 69 centers in 49 countries on six continents. This study comprehensively describes the characteristics of the 1,650 unique BRCA1 and 1,731 unique BRCA2 deleterious (disease-associated) mutations identified in the CIMBA database. We observed substantial variation in mutation type and frequency by geographical region and race/ethnicity. In addition to known founder mutations, mutations of relatively high frequency were identified in specific racial/ethnic or geographic groups that may reflect founder mutations and which could be used in targeted (panel) first pass genotyping for specific populations. Knowledge of the population-specific mutational spectrum in BRCA1 and BRCA2 could inform efficient strategies for genetic testing and may justify a more broad-based oncogenetic testing in some populations

Les droits disciplinaires des fonctions publiques : « unification », « harmonisation » ou « distanciation ». A propos de la loi du 26 avril 2016 relative à la déontologie et aux droits et obligations des fonctionnaires

The production of tt‾ , W+bb‾ and W+cc‾ is studied in the forward region of proton–proton collisions collected at a centre-of-mass energy of 8 TeV by the LHCb experiment, corresponding to an integrated luminosity of 1.98±0.02 fb−1 . The W bosons are reconstructed in the decays W→ℓν , where ℓ denotes muon or electron, while the b and c quarks are reconstructed as jets. All measured cross-sections are in agreement with next-to-leading-order Standard Model predictions.The production of $t\overline{t}$, $W+b\overline{b}$ and $W+c\overline{c}$ is studied in the forward region of proton-proton collisions collected at a centre-of-mass energy of 8 TeV by the LHCb experiment, corresponding to an integrated luminosity of 1.98 $\pm$ 0.02 \mbox{fb}^{-1}. The $W$ bosons are reconstructed in the decays $W\rightarrow\ell\nu$, where $\ell$ denotes muon or electron, while the $b$ and $c$ quarks are reconstructed as jets. All measured cross-sections are in agreement with next-to-leading-order Standard Model predictions